Least-squares Probabilistic Classifier: a Computationally Efficient Alternative to Kernel Logistic Regression

The least-squares probabilistic classifier (LSPC) is a computationally efficient alternative to kernel logistic regression (KLR). A key idea for the speedup is that, unlike KLR that uses maximum likelihood estimation for a log-linear model, LSPC uses least-squares estimation for a linear model. This allows us to obtain a global solution analytically in a classwise manner. In exchange for the speedup, however, this linear least-squares formulation does not necessarily produce a non-negative estimate. Nevertheless, consistency of LSPC is guaranteed in the large sample limit, and rounding up a negative estimate to zero in finite sample cases was demonstrated not to degrade the classification performance in experiments. Thus, LSPC is a practically useful probabilistic classifier. In this paper, we give an overview of LSPC and its extentions to covariate shift, multi-task, and multi-label scenarios. A MATLAB implementation of LSPC is available from ‘http://sugiyama-www.cs.titech.ac. jp/ ̃sugi/software/LSPC/’.